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ABSTRACT 

This paper uses a trial audit of history programs undertaken in 2011-2012 to ex¬ 
plore issues surrounding the attainment of Threshold Learning Outcomes (TLOs) 
in an emerging Australian national standards environment for the discipline of 
history. The audit sought to ascertain whether an accreditation process managed 
by the discipline under the auspices of the Australian Historical Association (AHA) 
could be based on a limited-intervention, "light-touch" approach to assessing 
attainment of the TLOs. The results of the audit show that successful proof of 
TLO attainment would only be possible with more active intervention into exist¬ 
ing history majors and courses. Assessments across all levels of history teaching 
would have to be designed, undertaken, and marked using a rubric matched to 
the TLOs. It proved unrealistic to expect students to demonstrate acquisition of 
theTLOs from existing teaching and assessment practices.The failure of the"light- 
touch" audit process indicates that demonstrating student attainment under a 
national standards regime would require fundamental redevelopment of the 
curriculum. With standards-based approaches to teaching and learning emerg¬ 
ing as international phenomena, this case study resonates beyond Australia and 
the discipline under investigation. 
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INTRODUCTION 

In the last decade, higher education systems around the world have become increas¬ 
ingly involved in the development of quality assurance and quality improvement frame¬ 
works to evaluate what students actually gain from a university education. In Australia, 
engagement with standards-based teaching at the tertiary level began with the recommen- 


Teaching & Learning Inquiry, Volume 3, Issue 2, pp. 89-105, 2015. 

Copyright © 2015 The International Society for the Scholarship of Teaching and Learning 


89 


• ••O 


• ••o 


Brawley, Clark, Dixon, Ford, Nielsen, Ross, Upton 

dations of the 2008 Bradley Review (Bradley Noonan, Nugent, & Scales, 2008, p. 136). 
Drawing on standards frameworks operating overseas and driven domestically by the 
2003 introduction of the Australian Universities Quality Assurance Agency (AUQAA), 
Bradley advocated for a new system for Australian tertiary education under which uni¬ 
versity programs would be taught to national standards with external evaluation as a di¬ 
mension of quality assurance. 

In 2010, history was chosen to participate in the Australian Learning and Teach¬ 
ing Council’s (ALTC) “Learning and Teaching Academic Standards Project” (LTAS) 
as a pilot discipline for the Arts, Humanities, and Social Sciences (Brawley et ah, 2011, 
p. 177-188; Brawley et ah, 2013a, p. 21-26). The project set out to explore how disciplines 
could respond to a national standards agenda by, in the first instance, designing what the 
ALTC called Threshold Learning Outcomes (TLOs). Within the Scholarship of Teach¬ 
ing and Learning (SoTL), “threshold concepts” are discipline-based hurdles that students 
must pass in order to move from one state of discipline understanding to another (Land, 
2011). According to Meyer and Land (2005, p. 373), such a shift in thinking is “trans¬ 
formative,” “irreversible,” and “integrative.” It is also likely that learning of such profound 
disciplinary significance may be “troublesome” (King & Felten, 2012, p. 5; Meyer & 
Land, 2005, p. 374). Indeed, Pace and Middendorf (2004) have observed that threshold 
concepts often significantly contribute to “bottlenecks” in learning. Once students have 
attained such discipline-specific threshold knowledge and skills, however, they may be 
deemed to have crossed a significant boundary: “ [T jhreshold concepts lead not only to 
transformed thought but to a transfiguration of identity” (Meyer & Land, 2005, p. 375). 
Such threshold concepts inform the construction of TLOs proposed by LTAS for courses 
or programs of study. 

Some argue that the adoption of TLOs as the basis of national standards has great 
promise because TLOs explicitly link disciplinary knowledge to teaching and learning 
practice (Hill, 2012, p. 53). This innovation, however, has also attracted criticism. Hussey 
and Smith (2002, p. 221-223) argue that the new focus on TLOs hinders authentic learn¬ 
ing and is symptomatic of an unwanted “managerialism,” in which the economic tail is 
wagging the educational dog. This misplaced focus produces a “monitored, audited and 
evaluated” educational process in which learning outcomes “are in danger of becoming 
little more than spurious devices to facilitate auditing at the expense of the educational 
process” (Hussey & Smith, 2002, p. 222). Hussey and Smith go on to distinguish be¬ 
tween explicitly stating student expectations to facilitate learning, and using learning out¬ 
comes to monitor educational practices in order to make education “more manageable” 
(2002, p. 223). Elsewhere, they argue that learning outcomes do not “lend themselves to 
strict auditing” because they “cannot be, in themselves, either clear or precise and do not 
specify objectively measurable entities” (Hussey & Smith, 2005, p. 232). Furedi (2013) 
goes much further, describing learning outcomes as “corrosive” because they cultivate a 
“utilitarian ethos to academic life” and reduce the “open-ended experience” of a univer¬ 
sity education (p. A2). 

A broad-based consultation process with history’s disciplinary community, led by 
Iain Hay for the ALTC, and coordinated through the discipline’s peak body, the Aus¬ 
tralian Historical Association (AHA), produced national TLOs for the bachelor’s level 
(“Australian Qualification Framework Level 7”). These TLOs were then endorsed by the 
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Table 1. Threshold Learning Outcomes for History 


Upon completion of a bachelor degree with a major in history, graduates will be able to 

Knowledge 

1. Demonstrate an understanding of at least one period or culture of the past. 

2. Demonstrate an understanding of a variety of conceptual approaches to 
interpreting the past. 

3. Show how history and historians shape the present and the future. (Subsequently 
revised by the AHA to “Show how history and historians shape their 
contemporary world.”) 

Research 

4. Identify and interpret a wide variety of secondary and primary materials. 

5. Examine historical issues by undertaking research according to the 
methodological and ethical conventions of the discipline. 

Analysis 

6. Analyze historical evidence, scholarship, and changing representations of the past. 

Communication 

7. Construct an evidence-based argument or narrative in audio, digital, oral, visual, 
or written form. 

Reflection 

8. Identify and reflect critically on the knowledge and skills developed in their study 
of history. 


AHA on behalf of the discipline. Subsequent to this endorsement, the AHA revised the 
eight TLOs for the discipline of history, which are reproduced in Table 1 (Brawley et al., 
2011, p. 171-188; Brawley et al., 2013a, p. 21-26). 

The purpose of the LTAS process was to create the TLOs, but it made no plans for 
compliance, implementation, and proof of attainment by students. As a result, endorse¬ 
ment of the standards proved to be an aspirational exercise, divorced from the realities 
of teaching and learning practice and separated from the problems of deployment and 
assessment. It appeared that the TLOs constituted a genuine expression of what study¬ 
ing history entailed. As such, it was assumed that they were already embedded in our daily 
practice as academic historians and teachers, and that students would therefore attain the 
TLOs as a natural outcome of existing history teaching. Such an assumption proved to 
be optimistic—and problematic. Having participated in this national standards project, 
and having produced TLOs broadly acceptable to the discipline, history next faced the 
challenge of establishing audit procedures to demonstrate that students in the nation’s 
history majors had successfully met the new standards. In 2011-2012, the After Standards 
project, funded by the ALTC (re-formed as the Office of Teaching and Learning [OLT] 
in 2011), designed and tested a “light-touch” process to gage student achievement of the 
history TLOs (Brawley, 2013a; Brawley et al., 2013b). This process was designed to work 
with existing history curricula in Australian universities. It was also intended to support 
a low-overhead, sustainable accreditation system that could be administered with limited 
resources by the discipline itself, under the auspices of the AHA. 

In this paper, we describe and assess that trial audit process as a case study in the 
challenges facing the imposition of teaching and learning standards in higher education 
everywhere. The trial audit revealed that although the TLOs were developed with the input 
and approval of the discipline, they proved aspirational in practice. The history TLOs did 
not reflect the realities of student learning in existing history majors. Without a thorough - 
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going revision of the history curriculum to embed standards consistently systematically 
and progressively into assessment tasks, demonstrating TLO attainment at the end of the 
final year of an undergraduate degree was difficult, if not impossible. 

Accordingly, we argue that if the Australian university sector intends to move to a 
standards-based environment where students can prove that they have met their discipline- 
specific TLOs upon graduation, then disciplines must undertake fundamental curriculum 
renewal, including significant training of—and buy-in from—academic staff. The most fea¬ 
sible way to ensure student attainment of the TLOs, moreover, is to incorporate a marking 
rubric based on those TLOs into assessment tasks that have themselves been built with 
the standards in mind. Student attainment of national standards, and any accreditation 
process demonstrating such attainment, must go hand in hand with curriculum renewal, 
assessment design, and staff engagement. 

THE AFTER STANDARDS PROJECT 

The After Standards project was led by historians from the University of New South 
Wales, the University of Queensland, and the University of New England, with support 
from the AHA and history academics in every relevant program in the country. The proj - 
ect s aim was to formulate a response to the emerging TLO-based standards environment. 
It would determine how TLOs should be embedded in teaching practice and audited, and 
provide the tools historians needed to engage and embrace the wide-ranging implications 
of standards-based teaching. The project organized a series of professional development 
workshops with plenary sessions and discussions for historians teaching across almost 
thirty Australian programs. The workshops also drew on the expertise of leading histo¬ 
rians from Britain and the United States who had confronted standards environments or 
who were internationally renowned for their work in SoTL (Brawley et ah, 2011, p. 191- 
193; Brawley et ah, 2013a, p. 24-25). 

It became clear at an After Standards proj ect workshop held at the University of New 
South Wales in April 2011 that compliance, especially proof of student attainment, was a 
maj or practical problem under a standards regime. A working party was formed to explore 
how the disciplinary community could develop its own audit and compliance process. In 
an internal report submitted to the AHA in May 2011, the working party recommended 
that the discipline develop a credible, nation-wide system of self-regulation. The report 
recommended that the AHA consider supporting an ex-officio board member with exper¬ 
tise in teaching and learning, a proposal that was accepted, endorsed, and implemented. 
The report also highlighted the positive face of the standards environment; it encouraged 
historians to reflect on teaching and learning practices and offered a framework for cur¬ 
riculum renewal. The Commonwealth’s move to establish a new national regulator the, 
Tertiary Education Quality and Standards Agency (TEQSA), added a sense of urgency 
to the report. The most difficult question was how to organize a credible system of self¬ 
regulation under a national standards regime. 

Although national, standards-based accreditation is new to the discipline of history 
in Australia, the idea of accreditation for other disciplines and professions is not. Profes¬ 
sions such as medicine, accounting, pharmacy, and engineering have long operated within 
the boundaries imposed by national accrediting bodies. Accreditation in these fields has 
been essential for students and universities, as well as for the international standing of 
the professions themselves. Moreover, internationally, much work has been done to con- 
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sider the feasibility of assessing learning outcomes and approaches to do so. The OECD 
Assessment of Higher Education Learning Outcomes (AHELO) and the US Collegiate 
Learning Assessment (CLA) process are two notable examples (Coates & Richardson, 
2012; Benjamin, 2013). 

ACCREDITATION 

The After Standards project developed and tested a self-regulated accreditation sys¬ 
tem for the discipline (Brawley et al., 2013a, p. 26-32). The questions driving the exer¬ 
cise included the following: 

1. Would it be possible for history, without experience of meeting external accredita¬ 
tion processes, to construct, support, and implement its own accreditation process? 

2. Could history use audit and accreditation to drive its own quality assurance and 
improvement processes within the new standards environment? 

3. Could history ultimately meet a hypothetical standards audit from a higher regula¬ 
tory body and thereby stave off potential control from an external accreditor such 
as a state agency? 

The three universities leading the project tested an accreditation system that in¬ 
cluded two components: a compliance phase and an audit phase. These institutions also 
took part in the trial. All three have strong research-intensive history departments and 
fine teaching reputations. The University of New England differs from the other two 
in that it has a large off-campus, distance-learning cohort and a larger number of non- 
traditional, mature-aged students. In the compliance phase, each university submitted a 
list of its courses, including stated learning outcomes and the assessment tasks aligned 
with them. The After Standards team then mapped that information against the TLOs. 
In most cases, this involved looking for a best, rather than a perfect, fit. A simple traffic- 
light system was utilized to align course-based learning outcomes with specific TLOs. 
The course-based learning outcomes were color-coded green for a good match, orange 
for a possible match, and red in cases where a learning outcome could not be mapped 
to a TLO (Table 2; colors replaced by indicative column). The goal was to construct a 
light-touch process, where standards-based accreditation sat atop current practices and 
made use of the current learning outcomes. 

The resulting tables were returned to the participating universities to confirm that 
the project team had properly mapped TLOs to course-based learning outcomes. This 
involvement compelled institutions to consider the degree of alignment of their courses 
with the TLOs. Discipline conveners were thus able to track how the TLOs were repre¬ 
sented across their own majors. Institutional course-level learning outcomes appeared 
to address most of the TLOs, with some exceptions. Most notably, TLO 8, which in¬ 
volves students reflecting on their studies as historians, and TLO 3, which deals with 
historians’ relationship to the contemporary world, were often poorly represented (see, 
for example, Brawley et al., 2013b, p. 38, Figure 11). These findings reinforce opinions 
expressed by After Standards workshop participants that TLOs 3 and 8 were the most 
difficult to teach and assess. 

With the exception of TLOs 3 and 8, the compliance trial confirmed that course- 
level learning outcomes were broadly aligned with TLOs. As a result, it was thought that 
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Table 2. An Indicative Example of Unit/course Mapping against the Learning Outcomes and the TLOs 


COURSE 

COLOR CODE 

LEARNING OUTCOME 

RELEVANT TLO 


Green (good match) 

Confront the relationship 
between past and present 

(3) Show how history and 
historians shape the present and 
the future 


Green (good match) 

Analyze textual material 

(4) Identify and interpret a wide 
variety of secondary and primary 
materials 

HIST3111 

Medieval 

Green (good match) 

Review a significant period 
of European history 

(l) Demonstrate an 
understanding of at least one 
period or culture of the past 

Europe 

Red (no match) 

Frame historical questions 



Orange (possible match) 

Recognize critical thinking 

(6) Analyze historical evidence, 
scholarship, and changing 
representations of the past 


Green (good match) 

Recognize logical 
argument and lucid writing 

(7) Construct an evidence-based 
argument or narrative in audio, 
digital, oral, visual, or written 
form 


programs were taught in such a way that students could demonstrate attainment of the 
TLOs, an optimistic conclusion that reflected a belief that the TLOs genuinely repre¬ 
sented the teaching practice of historians in Australia. The broad compatibility between 
local outcomes and TLOs revealed by our compliance trial obscured the fact that the 
alignment was often approximate and that TLOs did not always map to course assess¬ 
ments and curriculum. In short, the trial suggested that it would not be difficult for his¬ 
tory majors who attained local outcomes to “pass” an audit process based on the TLOs. 

In the audit phase of the trial, each university submitted five final (third) year as¬ 
signments from each of three nominated courses representing three randomly selected 
TLOs (chosen in light of the course learning outcome-TLO map). Capstone courses were 
chosen by default when they existed. The individual student assignments requested were 
the lowest passes from each course, an intentional decision driven by the pedagogical in¬ 
tention of TLOs (at the three participating institutions, a “pass” indicates a mark of 50% 
or higher in the course). Since TLOs represent minimum exit thresholds for all students 
who complete a major in history, we reasoned that the lowest pass students must be able 
to demonstrate that they had attained the TLOs. Assignments were de-identified so that 
neither the originating university nor the individual student was known to the audit 
team. Assignments were then distributed across the After Standards team and individu¬ 
ally marked against the nominated TLO for that assignment. The student work was not 
marked according to whether it should have passed or failed the particular assignment or 
course—many factors determine whether any given assessment passes or fails in those 
contexts—but instead, whether the students demonstrated that they had attained the 
specific TLO under consideration. 

To evaluate student attainment of TLOs, a four-point scale (0-3) was used. This 
scale was informed by the “common format” (1-4 scale) used in the Valid Assessment 
of Learning in Undergraduate Education (VALUE) rubrics, which were developed by 
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Table 3. Threshold Learning Outcome 3 Attainment Scale 


SCORE 

CRITERIA 

3 

Shows a critical understanding of the impact of historical events and processes on current 
and future situations. 

2 

Shows understanding of the impact of historical events and processes on current and 
future situations. 

1 

Shows a limited understanding of the impact of historical events and processes on current 
and future situations. 

0 

Shows little or no understanding of the current and future relevance of historical events and 
processes. 


the Association of American Colleges and Universities (AACU, 2013). The TLOs were 
used to create criteria for the marking scale. For example, TLO 3 (“Show how history 
and historians shape the present and the future”) was unpacked into four discrete levels 
of attainment (Table 3). 

Each assignment was allocated a score depending on the extent to which the corre¬ 
sponding TLO was met. For purposes of the trial, the After Standards team determined 
that for any given assignment, an average score of two was necessary to pass (i.e., a cu¬ 
mulative score of 10 points out of a possible 15 across all five assignments from a single 
institution marked against a given TLO). 

TRIAL RESULTS 

When the scores were tallied for all pieces of student assessment across the selected 
TLOs, the results were surprising (Table 4). Scores were uniformly low. Of the 40 pieces 
of assessment audited, five scored zero, indicating that the work did not at all meet the 
TLO against which it was examined. Eleven pieces “passed” with a two or higher, indi¬ 
cating that the TLO was substantially attained. Only one assessment scored a three, in¬ 
dicating that the TLO was deemed to have been wholly met. Only one course—the sole 
capstone course represented in the trial—warranted a (bare) “pass” (10 out of 15) in 
one TLO, meaning that the audited TLO was achieved. Even this single instance of suc¬ 
cess involved what is likely the easiest of the TLOs to meet: TLO 1, which requires the 
student to demonstrate some historical content knowledge. 

TRIAL DISCUSSION 

The results were singularly disappointing. If this exercise had been a genuine audit 
for accreditation, then no history program would have passed. We use the word “passed” 
advisedly because analyzing the data raised major questions, especially since we tested 
only a small sample of student assessments against the TLOs and used a sliding scale 
to make a judgment. What should constitute a “pass” for an assessment or a course in a 
standards environment? Were the results indicative of marking cultures that saw a bare 
pass as akin to a reward for effort? How many pieces of student work should be exam¬ 
ined per course in order to determine that a major had “passed,” i.e., to demonstrate that 
all students graduating with the major had attained all of the TLOs? Should all of the as¬ 
sessments submitted be required to have met the TLO in question to rate a pass or, for 
example, only three out of five? Should a major be judged against all TLOs at once to 
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Table 4. TLO Attainment Trial Results 


INSTITUTION 

COURSE 

TLO 

ASSESSMENTS 

RECEIVED 

EXAMINER 

ASSESSMENT SCORES 

AGAINST TLOs (THREE- 

POINT SCALE) 

TOTAL (OUT OF 

15; 10 TO PASS) 

PIECE 1 

PIECE 2 

PIECE 3 

PIECE 4 

PIECE 5 


X 

3 

Yes 

A 

1 

1 

1 

1 

0 

4 

University (A) 

Y 

6 

Yes 

B 

1 

1 

0 

1 

1 

4 


Z 

5 

No 

C 

_ 

_ 

_ 

_ 

_ 

_ 


A 

8 

Yes 

D 

0 

1 

0 

0 

1 

2 

University (B) 

B 

4 

Yes 

E 

o.s 

1 

1 

1 

1 

4.5 


C 

2 

Yes 

C 

1 

1 

2 

2 

3 

9 


M 

1 

Yes 

F 

2 

2 

2 

2 

2 

10 

University (C) 

N 

3 

Yes 

A 

2 

2 

2 

2 

1 

9 


O 

7 

Yes 

C 

1 

1 

1 

1 

2 

6 

AVERAGE 


6.1 


rate a pass, or should it be required to measure up against only a selected few (as in the 
trial)? Should different TLOs be tested each year? 

By the end of the trial, the After Standards team was convinced that if the discipline 
was to operate effectively within a national standards environment, then every audited 
student completing a major in history should have to pass every TLO in order to meet 
that standard. This approach was seen as necessary to endorse the major under audit 
against a comprehensive national standard. Further, all TLOs had to be audited at once 
rather than randomly assigned because not all TLOs are equal in difficulty. It is much 
easier, for example, to meet TLO 1 (“Demonstrate an understanding of at least one pe¬ 
riod or culture of the past”) than it is to meet TLO 3 (“Show how history and historians 
shape the present and the future”). If only a small number of TLOs were chosen for any 
given audit, it would be difficult to ensure that the selection of the TLOs over time was 
equitable for all majors participating in the accreditation process. 

The poor scoring in the trial also raised questions about the usefulness of employing 
a sliding scale. Since the TLOs represent minima, we concluded that such a scale was not 
an appropriate measure of attainment. It does not matter how well a student performs 
against the TLO for the purposes of accreditation, only whether that student met the 
TLO at all. A binary scale thus represents a more appropriate way to test this kind of at¬ 
tainment; the student should simply meet or not meet a given TLO. 

Since the TLOs are minima, furthermore, it was also inappropriate to allow some 
students’ exemplary achievement to counterbalance other students’ failure (e.g., threes 
on the sliding scale balancing out zeroes and ones to produce a “passing” average of two). 
For threshold standards to retain their meaning, all students had to meet all TLOs. 
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Most importantly, the trial revealed problems interpreting the TLOs for the purposes 
of judging whether assignments demonstrated their attainment. The standards themselves 
were problematic, too soft-edged for a metrics-based audit process, and in need of refine¬ 
ment and clarification. In this regard, our study supports the argument of Hussey and 
Smith (2002) that Threshold Learning Outcomes are difficult to audit because of their 
broad nature. Problems with the wording of the standards sat around two major areas: 
ambiguous standards and compound standards. 

Ambiguous wording obscured how markers should interpret what the standard was 
seeking; different markers could reach different conclusions about what the standard 
meant in the context of student work. The After Standards team decided that marking 
assignments would have been easier and more consistent if clear and unambiguous de¬ 
scriptors were provided for each standard. 

Another problem with TLO wording was that some of the standards evaluated more 
than one learning outcome. In other words, many of the TLOs are, in practice, compound 
standards. Auditors faced a problem with students who met one part of a standard but 
not another; should such performance be deemed as having met the standard? Com¬ 
pound standards forced a decision: apply a flexible aggregate of their multiple compo¬ 
nents, or require separate judgments on each aspect of the standard. For example, TLO 
4 asks students to “Identify and interpret a wide variety of secondary and primary ma¬ 
terials.” Must a student both identify and interpret? What happens if the student does 
not interpret but only identifies? Similarly, what happens if only secondary sources are 
identified or interpreted, without any reference to primary sources? For TLO 4 alone, 
some 15 permutations are possible. These practical problems with the TLOs made them 
difficult to deploy as benchmarks for authentic student work. The ramifications of teach¬ 
ing and testing them had not been adequately explored when they were developed and 
approved by the discipline. 

LESSONS FROM THE TRIAL 

Beyond flaws in our initial audit procedure and practical difficulties with the TLOs 
themselves, the trial produced a number of useful pedagogical lessons about modifying 
current practice to comply with the standards: 

1. Alignment. Mapping TLOs retrospectively to existing, local, course-level learn¬ 
ing outcomes is unlikely to be sufficient for audit purposes. Assessments that do 
not address the standards directly are unlikely to pass a compliance audit. Each 
major assessment task in a capstone or other final-year course must be designed 
to assess the TLOs. 

2. Curriculum development. Just as individual TLOs could not be met by existing 
assignments, ensuring that students meet all TLOs by the time they graduate re¬ 
quires redesigning courses and majors. Even if the TLOs represent historical teach¬ 
ing practice well, it is impossible to retrofit these standards onto currently existing 
courses and majors. Courses must be designed to address particular TLOs, and 
majors must be structured and coordinated so that all students practice all TLOs 
over the course of their undergraduate careers. A light-touch compliance process 
is unrealistic. Serious curriculum renewal will be necessary to meet standards. In 
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a standards-based environment, the standards must serve as the basis for the con¬ 
struction of the curriculum. 

3. Teaching the standards. Historians need to teach students—explicitly and trans¬ 
parently—how to meet the eight standards. 

4. Learning the standards. In order for student work to reflect attainment of the 
standards, the students themselves must know what the standards are and how to 
meet them. Transparency is again critical. 

TOWARDS A MORE ROBUST AUDIT AND ACCREDITATION PROCESS 

Taking onboard the lessons learned from the trial audit, th e After Standards team or¬ 
ganized a workshop in Darwin in November 2012, inviting eight exceptionally engaged 
historians from across Australia to formulate a new accreditation model (Brawley, et al., 
2013b, p. 52-54). The principles agreed on were as follows: 

1. All TLOs should be tested at once (each student’s work would be assessed against 
all eight TLOs). 

2. TLOs should be tested on a binary "meets”/ “does not yet meet” scale. 

3. All TLOs had to be successfully attained by all students. 

4. Review for accreditation should be double-blind (the reviewers would not know 
the institutions, and the reviewers would remain anonymous). 

5. Differences of professional judgment should be accommodated (e.g., for every 
TLO, at least two of three reviewers must agree that the standard is met/not met). 

6. A package of assessment needed to be developed for students to complete in the 
final year of their undergraduate program that could serve as a vehicle for them 
to demonstrate all TLOs, but one which could also be audited expeditiously for 
compliance and accreditation. 

7. Curriculum and assessment improvements had to be devised that would enable 
every student earning a major in history—even “low-pass” students—to consis¬ 
tently and reliably demonstrate attainment of all TLOs. 

Workshop participants considered three basic approaches to auditing and accreditation: 

1. A final examination approach. 

2. A multi-course portfolio approach. 

3. A capstone course output approach. 

A final examination approach, while useful in other contexts, was ultimately dis¬ 
carded as an inferior instrument to measure student attainment of the TLOs specific to 
history. While tests like the Collegiate Learning Assessment (CLA) are sophisticated 
and useful (Klein, Benjamin, Shavelson, & Bolus, 2007; USDE, 2006), they also have 
limitations (Kuh, 2006). The TLOs require analysis and understanding, proficient ex¬ 
pression, and reflection. Any test likely to demonstrate all of the qualities embedded in 
the TLOs would be extensive and onerous for students, institutions, and reviewers. The 
After Standards team remained unconvinced that extensive, formal examinations strictly 
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for accreditation would elicit authentic student work. Instead, the team feared that a 
formal examination would be treated as an arbitrary burden (cf. “The VALUE Project 
Overview,” 2009; Rhodes, 2008). 

Strong support emerged from the workshop for either an assessment package pro¬ 
duced by students in a major capstone course or a portfolio of student work. The capstone 
model involved work from a single course, while the portfolio model required students 
to keep samples of work undertaken over a period of time and across multiple courses. 

Delegates at the workshop recognized the advantages of the portfolio option. 
Portfolio-based assessment has worked well in other disciplines (O’Sullivan et al., 2012; 
Rowley & Cunbar-Hall, 2009). ePortfolios have also attracted considerable interest in 
Australia as a source of evidence for student learning and part of transition into employ¬ 
ment (Hallam & Harper, 2008), and in the US they form the basis of the VALUE process 
(Rhodes, 2008). Use ofportfolios to evaluate TLO attainment presupposes that the TLOs 
will be taught and developed across a range of courses. This system would encourage pro¬ 
gression but at the same time allow different courses to concentrate on delivering par¬ 
ticular TLOs appropriate to their aims and content. Not all courses would need to cover 
all TLOs. Students could compile their portfolios from a range of work, encouraging 
diversity of assessment tasks and requiring students to engage explicitly with the TLOs. 
Certainly the traditional essay would be part of the package, but it might not be the only 
type of task undertaken by students. The portfolio approach, however, places the greatest 
administrative time and resource demands on students, the institutions, and compliance 
review committees. It requires that processes be in place to collect and manage samples 
of student work across many courses. There are also logistical problems for students who 
change majors or universities. 

Auditing capstone course outputs was seen by others as a less burdensome approach 
than the use of portfolios, as well as one more suited to the culture of the discipline than 
a cumulative exam. Capstones are well-established in the United States and Britain, and 
are becoming more common in Australia (Hauhart & Granhe, 2010; van Acker, 2011). 
A capstone is a course taken in the last year of major study, ideally as the ultimate course 
in a major or program. Capstones bring together all of the learning that has taken place 
over the duration of the major, often culminating with a student-led research project 
(e.g., Greenbank & Penketh, 2009). Learning outcomes or graduate attributes have been 
successfully embedded in capstones (van Acker, 2011), making them a likely venue for 
assessing the TLOs in our context. At the Darwin workshop, advocates of this approach 
argued that a major research project with a reflective exercise in the capstone course 
would prove an efficient means of assessment, while recognizing the potential for a wider 
variety of assessment. It was agreed by all that it would be possible to design an assess¬ 
ment package for a capstone course that would allow students to address all TLOs. Such 
a package would be easier to collect, distribute, and audit against the standards than a 
multi-course portfolio. Capstones, however, present staffing challenges. Can we ensure 
that all students have a supervised research experience in the capstone when providing 
it is so labor-intensive ? Can we accommodate students’ desire to study particular periods 
or cultures ? Although probably less administratively onerous than a portfolio, workshop 
participants recognized that the staffing resources required for a successful capstone may 
place it out of reach for some institutions. 
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The workshop resolved that any decision to institute either a capstone or a portfolio 
approach within an individual major must be determined by the peculiar requirements 
of each university’s systems, structures, and student cohort. By the end of the workshop, 
delegates had decided to leave the specific assessment package compiled by students for 
TLO audits to individual institutions, whether from a single course or a range of courses. 
Whatever package was submitted by an institution would, however, be marked against a 
standard rubric based on the TLOs. Rubrics have the advantage of standardizing mark¬ 
ing of, for example, e-portfolios (Tomkinson & Freeman, 2007). The VALUE project in 
the US has demonstrated that rubrics, as statements of expected learning, are effective 
at implementing shared core criteria across diverse institutions. They can also improve 
teaching and learning by providing students and instructors with clear, shared expec¬ 
tations, and are more likely than standardized tests to capture authentic student work 
(Burnett & Williams, 2009; Rhodes, 2008; Rhodes, 2009; “The VALUE Project Over¬ 
view,” 2009;). While the VALUE project focused on the combination of rubrics with 
e-portfolios, After Standards workshop participants believed that they could be equally 
applied to outputs from a single course, particularly a capstone (Levia & Quirring, 2008). 
An output-agnostic, rubric-based approach offers a number of advantages. Most impor¬ 
tantly, it provides a uniform minimum standard to over 30 history programs in Australia, 
all of which have different identities, student cohorts, and institutional settings, while also 
recognizing the autonomy of each. 

With the assistance of some preliminary work on TLO “descriptors” drafted by the 
original LTAS project and further refined at the Darwin workshop, the After Standards 
team produced an expanded “National Rubric” that specified what was required to meet 
each TLO, including additional guidance to facilitate local implementation (Brawley, et 
al., 2013b, p. 93-94). Compound standards were also disaggregated, with the expectation 
that students must pass each component of the TLO in order to pass the TLO as a whole. 
This rubric was designed to ameliorate some of the practical problems encountered with 
the bare standards. It can be applied to a wide range of student outputs, from a single class 
or from courses across an entire major. The rubric unpacks the TLOs, helping academic 
staff to understand what must be taught and students to understand what must be learned. 

A natural extension of the rubric involved suggesting “credit,” “distinction,” and “high 
distinction” levels of attainment (corresponding to the grades typically assigned in Aus¬ 
tralia). Although the After Standards team agreed that any attainment above a “pass” was 
not directly relevant during an audit, and that individual institutions should be free to 
define their own criteria for higher levels of attainment, the production of model mark¬ 
ing rubrics could encourage uptake and deployment. To initiate this process, the After 
Standards team also developed a template suggesting what such a local implementation 
might look like (Brawley et al., 2013b, p. 95). Considering the failure of the first trial 
audit, we considered such deployment crucial to ensure alignment between coursework 
and the TLOs, which is, in turn, required for students to achieve them and for majors to 
pass audits against them. 

Finally, the After Standards team created a sample audit process built around the ru¬ 
bric and accommodating the variety of output discussed above (see Appendix A). Since 
all agreed that the TLOs genuinely reflect practice for Australian historians, in this case 
“teaching to the test” represents authentic learning and teaching, leading to outcomes 
endorsed by the discipline. 
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STATUS AND OUTLOOK 

The implementation of a standards regime for history and other disciplines has been 
delayed in Australia (Brawley et al., 2013a, p. 21-24). Nevertheless, the community that 
has coalesced around the After Standards project has pressed ahead, in the belief that na¬ 
tional standards are coming and convinced that the discipline must develop its own plan 
to face them. The accreditation model described above and in the appendices will now 
be considered by the AHA. 

If the AHA is to supervise the accreditation of history majors, much preparatory work 
is needed. Historians across the country who have already endorsed the TLOs need to 
subscribe to the concept of an accreditation process. Curricula, courses, and assessments 
need to be redesigned with the TLOs in mind. Administrative systems must be put in 
place to select, collect, de-identify, and process assessments for audit. Review panels must 
be recruited and organised. Provisions must be made to improve majors whose students 
do not meet the TLOs, and therefore fail the audits. 

Whether the AHA and individual stakeholders—institutions and majors—will em¬ 
brace an accreditation model that requires broad and deep curriculum renewal remains 
to be seen, but if the discipline does not have an accreditation process ready to deploy 
when national standards are introduced, a far less desirable one maybe imposed upon it. 

CONCLUSION 

A national standards regime is emerging fitfully for the history discipline in Aus¬ 
tralia. National standards offer an opportunity for the discipline to articulate the nature 
and value of what we teach, as well as to demonstrate student learning to our communi¬ 
ties, institutions, and regulatory bodies. The After Standards project has spearheaded a 
proactive effort to devise discipline-led accreditation while also improving the quality 
of history teaching and learning. 

Initially, the After Standards team, in consultation with our stakeholder community, 
proposed a “light-touch” approach to compliance auditing. A light touch proved insuffi¬ 
cient, however. As a result, the After Standards project revisited potential audit approaches. 
A rubric has now been developed to guide educators through curriculum review and au¬ 
ditors through the assessment of student work for TLO attainment. This rubric can be 
applied to any reasonable package of student assessments, from a single course or from 
multiple courses, respecting institutional circumstances and autonomy while measuring 
compliance with national standards. The rubric, in expanded or modified form, can also 
be deployed for marking and evaluation of routine student work at individual institutions 
in order to ensure alignment with the TLOs across assessments, courses, and curricula. 
Such alignment is crucial. Successful, measurable attainment of the TLOs will require it, 
along with additional curriculum renewal to ensure that all students majoring in history 
attain the TLOs during the course of their undergraduate careers. Since the discipline 
community has endorsed the TLOs as representative of authentic practice, their deploy¬ 
ment offers significant opportunities for not just quality assurance and compliance, but 
also for quality improvement—more effective teaching and more profound learning of 
history in Australia. 
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APPENDIX A: MODEL ACCREDITATION PROCESS 

Principles 

• Accreditation will be based upon an assessment package (e.g., a research essay plus 
associated reflective work from a capstone, or a student portfolio). 

• If work is collected from a single course, the course selected should be third year, 
preferably a capstone, preferably offered in the final semester of the student s career. 

• If work is collected as a portfolio, it may come from any combination of courses, 
but should not exceed XXXX words in length (maximum length TBD). 

• Each assessment package submitted for review will be tested against all eight TLOs. 

• Each TLO will be evaluated on a binary “meets/does not yet meet” scale, using 
descriptors encapsulated in a rubric. 

• The rubric should be made publicly available so that student assessments / course 
learning outcomes / major learning outcomes / etc., can be aligned with it. It is 
recommended that student assessments be marked using a customized version of 
the supplied full marking rubric. 

• Review for accreditation is double-blind—the reviewers do not know the institu¬ 
tions, and the reviews will be anonymous. 

• Differences of professional judgment will be accommodated. For every TLO, at 
least two of three reviewers must always agree that the standard is met. 

Preparation 

1. Institutions should avail themselves of the publicly available rubrics (“meets/does 
not meet” and full marking rubrics). 

2. Institutions must be prepared to provide a package of assessment from students 
(maximum length XXXX words). This package may, for example, consist of a port¬ 
folio of student work or the outputs from a capstone course. 

3. Institutions must provide sufficient administrative support—clean, anonymized 
submission of the five lowest pass assessment packages within a specified time 
frame. Students should also be informed that their assessments may be reviewed 
for accreditation purposes (anonymously). 

a. If the institution submits a single assessment, “lowest pass” means the lowest 
pass on the task. 

b. If the institution submits a package of multiple assessments from a single course, 
lowest pass means the lowest passes for the course. 

c. If the intuition submits a student portfolio drawing from multiple courses, low¬ 
est pass means the lowest GPA or equivalent amongst all students graduating 
in the major in the year audited. 

4. The process will be conducted online—a Google Apps model is presented below. 
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a. The accrediting body (e.g., the AHA) opens a Google Apps account (free or low 
cost) so that we can have a “branded” Google Drive, email addresses, etc. 

b. Reviewers should be given email addresses associated with this account (for 
security reasons, can be forwarded). 

c. A Google Doc of the TLO rubric with full descriptors will be posted on the 
Google Docs account for reviewers’ information. 

d. A Google Survey based on a brief version rubric (without descriptors) will be 
used to collect and tabulate results. 

e. Output will be as spreadsheets (full results and summary). 

Process 

1. Institutions upload clean, anonymous copies of the five lowest pass assessment 
packages to a secure, online folder. 

2. Reviewers are given a link to two folders: one with the rubric, one with the sub¬ 
mitted assessments. 

3. Reviewers prepare by examining the “elaborated” rubric with all of the descriptors 
and “meets/does not yet meet” criteria. 

4. Reviewers read each assessment and score it. Assessments and institutions will be 
identified only by number for tracking. 

5. A report will be generated with each institution given a top-line “meets”/”does 
not yet meet” grade, plus the details of how each reviewer marked each TLO 
(anonymous). 
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